fosmfandomcom-20200213-history
Instructions for installing the server on Debian/Ubuntu/Mint
These instructions to install the Fosm API server were developed for Linux Mint 18, but should work on modern Debian/Ubuntu systems. Some tweaks would be required for other distros and systems. You need admin privileges to perform several of the steps. It is recommended you have plenty of hard drive space: the empty database takes about 300Gb. The populated fosm database is around 700Gb (as of 2016). Using a web browser, download the GT.M database from: https://sourceforge.net/projects/fis-gtm/files/GT.M-amd64-Linux/V5.5-000/ and get the file: gtm_V55000_linux_x8664_pro.tar.gz This guide explains how to install the system into the /srv/fosm path, but this can be changed if needed. The software will be installed under the www-data username and group, because it allows editing by multiple authorized users and the server is largely web based anyway. This guide uses nano as an editor, but any other will do just fine, such as xed, pluma, gedit, emacs, vi, etc. Add yourself to the www-data group: sudo nano /etc/group Scroll down. Add your linux username to the end of the line that is like: www-data:x:33: Save the file (in nano, do ctrl-o, enter, then ctrl-x), log out and log back in. Check you are in the www-data group by the command: groups Extract and install the GT.M database software: cd ~/Downloads/ mkdir gtm tar -xf gtm_V55000_linux_x8664_pro.tar.gz -C gtm cd gtm sudo mkdir /srv/fosm sudo ./gtminstall --installdir /srv/fosm/gtm55 --user www-data --group www-data cd .. sudo rm -R gtm Now we start creating the folder structures needed for the server. (Note: the next section, gtmx does not seem to be necessary, just create an empty folder by "mkdir gtmx" instead.) cd /srv/fosm sudo chown www-data:www-data . sudo chmod ug+rwx . mkdir planet mkdir planet/minute-replicate/ mkdir data mkdir serenji git clone https://github.com/fosm/fosm-scripts.git scripts git clone https://github.com/fosm/gtmx.git git clone https://github.com/fosm/fosm.org-webpage.git ln -s gtm55 gtm ln -s gtmx gtm55x ln -s ../scripts/xapi.gld data/xapi.gld ln -s fosm.org-webpage/fosm.org www mkdir /scripts/o sudo chown www-data:www-data -R . sudo chmod ug+rwx -R . If you are installing to a custom path, check the first line of /scripts/profile and set the path you are using. GT.M requires several environment variables to operate correctly. Assuming you are using bash, add the following lines to the end of your ~/.bashrc file: source /srv/fosm/scripts/profile export EDITOR=/bin/nano Configure the database to use 8k database blocks instead of the default 4k blocks: edit /etc/sysctl.conf (using sudo because it is protected) and add the following lines: (NOTE: is this necessary? Is a reboot needed to apply this?) # XAPI settings for 8k database blocks kernel.shmmax = 67108864 kernel.shmall = 67108864 Note: the fosm.org server uses kernel.shmmax=4294967295 ... why? You should be able to initialize the database at this stage. GT.M should always be run from the folder containing the database i.e. /srv/fosm/data cd /srv/fosm/data mupip create The system might churn the hard drive at this stage to pre-allocate database space. You can then import a planet dump file into the database. The planet file must use single quotes for xml tags, not double quotes. It also probably requires each tag to be on a separate line (TBC). (NOTE: This process doesn't work! However, it is possible to skip this and continue with a blank slate database.) cd scripts mumps loadPlanet.m cd ../data mumps -run ^loadPlanet cat ../hereford.osm > planet.pipe The apache web server is used to process HTTP requests from the users browser. It also passes API requests to the GT.M scripts. These instructions assume apache 2.4. (fosm.org is currently on version 2.2.9) sudo apt install apache2 cd /etc/apache2/mods-enabled sudo ln -s ../mods-available/cgi.load sudo ln -s ../mods-available/rewrite.load sudo rm /etc/apache2/sites-enabled/000-default.conf cd /etc/apache2/sites-enabled/ sudo ln -s /srv/fosm/scripts/fosm.conf fosm.conf Edit fosm.conf and set a unique password for SERVER_LINK_PASSPHRASE. This password is used by apache to access the private GT.M backend service. If you wish to run the server on a custom port, both fosm.conf and /etc/apache2/ports.conf must be changed. Restart apache: sudo service apache2 restart Use your web browser and look at http://localhost/ to check apache works. If that fails, start by checking: tail /var/log/apache2/error.log The website needs more configuration to talk to GT.M. We also need to prepare the xapid service. cd /srv/fosm/www rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi ln -s /srv/fosm/planet/ planet cd /srv/fosm/www/api/0.6 rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi cd /srv/fosm/www/api/0.6/user/ rm serverLink.cgi ln -s /srv/fosm/scripts/serverLink.cgi serverLink.cgi cd /srv/fosm/scripts make cd /srv/fosm/data We also need to set some variables within GT.M. Run: gtm Then issue the following commands (note: the specific values are probably absurd but they more or less work): s ^osmPlanet("instance")="xapi" s ^osmPlanet("date")=20140101 s ^osmPlanet("osmSession")="bbb2559f0e425315329ca1f010197858" s ^serverLink("REST")="rest" s ^serverLink("port")=6530 s ^serverLink("logLevel")=0 s ^date("timezoneOffset")=8 s ^munin("apiCalls")=27597139 s ^munin("responseDB")=84932980 s ^munin("responseIO")=10466660 s ^munin("responseTotal")=141848508 s ^id("changeset")=1000074436 s ^id("node")=1000594558245 s ^id("pendingUid")=100000383 s ^id("relation")=1000000153460 s ^id("uid")=100000156 s ^id("way")=1000002478076 The server password also needs to be set the same as SERVER_LINK_PASSPHRASE in the apache configuration. d setup^serverLink Enter a password, enter any server description string, then press enter when it asks for a second password. You can check the current values of these parameters with commands like: zwr ^osmPlanet zwr ^serverLink WRITE ^serverLink("REST") Exit GT.M by typing "halt" or "h" and pressing enter (or ctrl-d). Start the xapi services with the commands: cd /srv/fosm/data export GTMCI=/srv/fosm/scripts/xapi.ci ../scripts/xapid This should just run without returning to the command line. Just leave this run. Other commands can be performed in a different terminal window. You can also run the script fosm_up in the scripts directory. This will return to the command line. cd /srv/fosm/scripts ./fosm_up You can confirm the service is running using: netcat 127.0.0.1 6530 If it returns immediately to the command line, the service is not running. Just press enter for netcat and the TCP connection is terminated (by the GT.M server, I think). Next, try doing a basic map query: http://localhost/api/0.6/map?bbox=-2.742031,52.0509012,-2.7326135,52.0567448 If you have permission problems, try: cd /srv/fosm sudo chown www-data:www-data -R . sudo chmod ug+rwx -R . Look at: tail /var/log/apache2/error.log If you still can't connect to GT.M cd /srv/fosm/scripts export SERVER_LINK_IP=127.0.0.1 export SERVER_LINK_PORT=6530 export SERVER_LINK_PASSPHRASE=your-password-here ./serverLink.cgi That should produce no errors. Incidentally, API errors are logged into the database and can be retrieved by the GT.M command: zwr ^error API calls and responses are logged here: zwr ^log Crontab Set cron to run scripts automatically: sudo crontab -u www-data -e to contain: #* * * * * /fosm/scripts/loadDiff * * * * * cd /fosm/scripts; source exportDiff #50 * * * * /fosm/scripts/purgeJournal # Run the planet dump monthly on 1st of month at 5am #00 05 1 * * /fosm/scripts/planetDump Create a user account We can create a user account but bear in mind email sending is probably not configured. You cannot directly use the email confirmation link. However, we can work around this. Register a new user with a web browser: http://localhost/user/new Then on the command line: cd /srv/fosm/data gtm Then issue the command: zwr ^pendingUser Look for the "emailToken" parameter. Use a web browser and go to the URL: http://localhost/user/confirm/emailToken with the "emailToken" replaced by the actual value. That should enable the user for editing. You can now start JOSM, change the server URL to: http://localhost/api and start editing!